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ABSTRACT 

Children with serious emotional disorders receive 
inadequate and inappropriate services, rather than the comprehensive 
and coordinated system of care needed. To develop the knowledge 
needed to provide adequate and appropriate services, an alliance is 
needed among advocates, practitioners, and researchers. The Mental 
Health Services Program for Youth, supported by the Robert Wood 
Johnson Foundation, is a demonstration program intended to develop 
comprehensive and coordinated systems of mental health care. The 
community-level projects supported by the program develop such 
services as case management, emergency services, respite care, and 
therapeutic foster care. Each project incorporates a plan for 
integrating services and financial resources for children served by 
multiple agencies and service systems. A quasi-experimental design is 
being used to evaluate the program, by context, process, and outcome 
and by organizational, financial, and client characteristics. Thf 
evaluation design involves an interview-based organizational 
assessment, use of a management information system to assesr client 
data, and a case conference involving input from all members of the 
clinical team. The importance of bridging the gap between theoretical 
and empirical knowledge to improve the current delivery system is 
stressed, (c tables/figures) (JDD) 
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Several weeks ago, Rich Hunter (Symposium Chair) shared with me a draft of his 
introduction. The original draft included several quotes from a "state of the art" review of 
mental health services for children (Tuma, 1989) that appeared in the 1989 Am&ican 
Psychologist Special Issue on Children. I had an odd sense of deja vu as I read the quotes 
and I was prompted me to read the article carefully. To my surmise and chagrin, I 
discovered mat much of the article (228 of 342 sentences) was taken verbatim, or with 
only minor editing, from a monograph I had written in 1986 with Ted Cross and several 
colleagues (Saxe, Cross, Silverman, with Batchelor, 1987) for the Congressional Office of 
Technology Assessment (OTA). The OTA monograph had been summarized in a public 
policy forum article for American Psychologist that had appeared only months before the 
Special Issue. 

It is not my intent to digress from a focus on the important issues that face children 
with serious emotional disorders to deal with a purely adult problem. For the moment, I 
only want to provide a footnote for the record of what 1 discovered and, by doing so, 
share a private burden. It has been the occasion for much thought about the ideas that 
underlie what we advocate for children. In an ironic way, my discovery of this adult 
problem has relevance for what I would like to say this morning about children's mental 
health policy. 

The thrust of our original monograph for OTA (and, by extension, the AE article) 
was that children with serious emotional disorders receive inadequate and inappropriate 
services and that we need to develop comprehensive and coordinated systems of care. 
These ideas are not necessarily original and have been the theme of several major national 
reports (e.g., President's Commission on Mental Health, 1978). NIMH - through the 
Child and Adolescent Service System Program (CASSP) - has been working to implement 
these ideas. Pending federal legislation (The Children's and Communities' Mental Health 
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Systems Improvement Act) will mate them a visible feature of national policy. The view 
that our fragmented system of care for children needs to be rationalized and made 
comprehensive has become a fundamental tenet Comprehensive and coordinated care 
have become the catch-words adopted by advocacy organizations from the APA to the 
National Mental Health Association. That the way of expressing these ideas would be 
adopted without attribution is, perhaps, only one indication of their widespread acceptance. 

Unfortunately, between die rhetoric of advocates for a new system of children's 
mental health services and the reality of practitioners providing effective and efficient 
services, lie a number of unknowns. What is the actual magnitude of children's mental 
health problems? How is it possible that one out of five children has an emotional 
disorder that requires treatment? What do we really mean by comprehensive services? 
What are the limits to what society can and should do to assist troubled children and 
families? What types of interventions are most effective for what children? Are all Axis I 
diagnosed children equal in terms of their needs for services? What types of interactions 
are there between diagnoses and conditions such as ooverty? 

Until recently, mental health policy afficiandos - in particular those interested in 
the welfare of children - had to contend with a different dilemma. Existing knowledge 
about children's mental health problems and treatment was not being used (see Knitzer, 
1982; Stroul & Friedman, 1986) - even in a rudimentary way. We labeled this (Saxe, 
Cross, & Silverman, 1988) the "gap between what we know and what we do". But, the 
1980s have given way to the 1990s. The 1990s are becoming the era of change for health 
and social services. One way or another, our health and social service institutions will be 
changed and children's mu.tal health care will not be immune from change. The only 
question is who ill influence what type of system emerges. If we want it to reflect the 
best that the science and practice of psychology can offer, it is essential to understand how 
to operationalize the concepts of advocates for children. Our urgent task is to develop a 
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process whereby we can develop, test and implement new ideas about children's mental 
health care. 

To implement new ideas is a great deal more demanding than developing consensus 
or gaining allies behind the right catch-words and phrases. Developing new systems 
involves translating principles into practice. The depth of our commitment to children and 
our individual skills as practitioners will be for naught unless we can develop the 
knowledge needed to provide adequate and appropriate services. To develop this 
knowledge base, an alliance is needed among advocates, practitioners, and researchers. It 
will, undoubtedly, be an alliance fraught with tension, but hopefully a healthy tension. 

The tension is a result of differences in perspective. Advocates need to see 
problems in simple ways and upbraid those who diminish the needs of children. 
Practitioners must exude confidence about their interventions and avoid the burden of 
skepticism about the efficacy of care. Researchers must maintain a skeptical frame of 
reference and view all ideas as tentative hypotheses whose acceptance is merely a 
confirmation rather than a proof. Nevertheless, for ideas to get to the stage where practi- 
tioners can innovate with them and new knowledge be developed, collaboration is 
required. If we are to improve the chaotic state of care for children with emotional 
disorders, there is really no alternative to our joining forces. Today, I would like to 
describe one effort of advocates, practitioners and researchers to join together to develop 
new models of care for children with serious emotional problems. My father used to tell 
me "Wait 'till you get older and learn how much you need to learn". I hope from my 
discussion today it will be clear that we have much to learn about children's mental health 
care, but that we have finally launched some important efforts to reform a dysfunctional 
system. 

Robert Wood Johnson Foundation Demonstration 
Much has changed in the 5 years since I worked on our report for Congress on the 
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state of children's mental health care, but it is dear that the fundamental problems of lack 
of access and appropriate care have not been solved. Some children are probably receiv- 
ing more appropriate care, and the growth of psychiatric hospitals has been restrained 
(perhaps by reducing the average length of stay), but it is not dear that we have made the 
fundamental changes necessary to provide appropriate universal care. In a recessionary 
economic period where federal, state, and local governments play a game of "hot potato" 
over responsibility for health care and social services, it may be ddusionary to believe that 
much could be accomplished. 

One exciting development, however, is a series of demonstration projects now 
being conducted to test new ways to provide services. Demonstrations are being carried 
out by several states, as well as by the Department of Defense. One of these 
demonstrations, the Mental Health Services Program for Youth ;MHSPY), is supported by 
the Robert Wood Johnson Foundation (RWJF). The Foundation wants to demonstrate that 
services for children and youth with serious mental illness can be improved through 
comprehensive and coordinated systems (Beachler, 1990). RWJF is providing 
approximately $2.5 million to each of 8 states over the next 4 years to aid them in 
improving the system of care for seriously emotionally disturbed children. States will use 
Foundation funds to supplement existing resources and will make structural changes in the 
organization and financing of mental health services. As part of this demonstration, we 
are conducting an independent evaluation of the initiative. 
Goals 

The RWJF demonstration is intended to support model systems of care in 
geographically and socio-economically disparate communities (Figure 1). It is a project 
that resulted from the work of advocates for children, was designed by practice experts, 
and includes a research component. The project probably fails to meet fully the needs * 4 
each these groups - the funding is not large enough to make a substantial difference, 
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practitioners have Utile flexibility, and as a research effort the potential lor unambiguous 
assessment of hypotheses is limited. Nevertheless, it is a model of collaboration and a 
prototype of how ideas about improving care for children with serious emotional disorders 
can be implemented and tested in actual settings. 

The RWJF initiative is designed to encourage state and local community 
innovation. States are the recipients of RWJF funding, but the projects are being carried 
out at the community level (depending on the site, the operating agency may be a unit 
either of state or county government). The component services of the RWJF projects are 
varied (See Table 1). They range from case management and emergency services (features 
of all projects) to day treatment and therapeutic roster care (explicit features of most 
projects) to respite care and transition services for adolescents (present in only some 
projects). Each project also incorporates a plan for integrating services and financial 
resources for children served by multiple agencies and service systems. And each project 
proposes to restructure how services are financed. Funding is perhaps the glue that ties all 
of us concerned with mental health for children together. 

My goal, as evaluator, is to provide objective, usable information about the 
demonstration - for the projects and Foundation, but ultimately for the wider audience of 
those who make decisions about mental health policy. The raison d'etre of the research 
study is to provide information that will facilitate replication of effective programs. 
Although the logic of such research is no different than that of any type of research, we 
have a special obligation to identity the contextual factors that underlie program 
effectiveness. Historian of psychology E. G. Boring remarked once that "today's 
confound becomes tomorrow's independent variable". The evaluation task is to understand 
the confounds that interfere with the effectiveness of theoretically effective care for 
children with mental health problems. 

The key construct being tested by the demonstration is that provision of 
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comprehensive and coordinated mental healtii care is more effective and efficient than 
more fragmented systems of care for seriously emotionally disturbed children. 
"Comprehensive care" has been operanonalUsed by the projects as the implementation of a 
"continuum of care" that encompasses a broad range of services, "Coordinated care" is 
operationalized by the provision of case management services and implementation of joint 
or interagency structures and working agreements. Each site is implementing these 
constructs somewhat differently; iinprovement in the quality of care is anticipated as an 
outcome of a more comprehensive, coordinated service system. 

Although the projects supported by the RWJF initiative share a common conceptual 
base, states and communities differ substantially in their governmental structures, in their 
financial operations, and even in the problems faced by their children. Thus, we need to 
collect a considerable amount of idiosyncratic data about each site and be as descriptive as 
possible. Our primary task is to identify the components of comprehensive, coordinated 
child mental health systems and their relationship to service provision and costs. Second, 
we need to understand the impact of systems changes on children and their families. 
Methodology 

Knowing what we need to understand and developing appropriate methods are 
somewhat different problems. For the researcher, this is the equivalent issue to translating 
ideas about mental health care into actual treatment interventions. Unfortunately, available 
tools to assess mental health systems are derived from clinical research and the assessment 
of individual change in psychopathology. The RWJF demonstration requires that we 
collect information at a different level and that we integrate clinical data with 
organizational data. 

To guide the research, we have identified a series of research questions (see Table 2). 
The questions are organized by context, process, and outcome and by three "domains of 
inquiry" - organizational, financial, and client. We need to understand the nature of the 
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systems, the resources available, and the clients. Eventually, we need to link answers to 
these questions to outcomes, hopefully, through understanding the mediating variables 
(process questions). It is undoubtedly a complex way of thinking about the project, but 
providing comprehensive and coordinated care is inherently complicated 

The evaluation design is quasi-experimental, with comparisons among the sites and, in 
some cases, to non-project sites as the essential approach. But, the far more difficult 
problem, given the nature of the demonstration, is how to measure the dependent 
variables. Our decision is to use multiple methodological strategies; in essence, to collect 
data from a number of different perspectives. Currently, mis involves: (a) An interview- 
based organizational assessment; (b) use of a management information system to assess 
client data; and (c) a case conference involving input from all members of the clinical 
team. 

Organizational assessment. The first design component uses structured interviews 
and document review to collect data to describe the organizational and financial systems at 
each site. In-depth case studies of each of the eight projects folly funded by RWJF and 
more limited case studies of the tour sites that received only initial implementation funds 
will be developed. Each case study has several goals (see Table 3): (1) to describe the 
environment for children's mental health services, (2) to document the structure of the 
service system, (3) to describe the nature of interventions to be supported by each project, 
(4) to document the sources of funding for the range of services, (5) to describe the 
specific client population, and (6) to chronicle the changes in coordination, provision and 
funding of services that are the result of project implementation. 

Management information system. One of our innovative efforts to collect usable 
data consistent with the needs of the system is to take advantage of the ^formation 
collected routinely as part of project management and accountability. Although each state 
(and, in some cases, each agency) maintains different record-keeping systems, they all 
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collect extensive client-based data on clinical encounters and finances. In collaboration 
with the MHSPY National Program Office (Drs. Cole and England), we are developing a 
sophisticated computer-based system for such record-keeping. 

The Management Information System is called SEDTRACK, with a short version 
dubbed QUICKTRACK on the way (see Table 4). When fully implemented, it will allow 
a case manager to enter information about a child and family into a PC. SEDTRACK will 
give case managers a powerful relational database program to organize the disparate 
information about children and families being served and make it possible to update easily 
records and case plans. There is nothing simple about this program, but the idea is that 
managing complex cases is hindered by manual record keeping systems and is actively 
frustrated by inflexible linear record keeping systems. 

From a research perspective, the advantage of SED/QUICKTRACK is that all of 
the information about clients and families will be available for analysis, without duplicative 
assessment and coding. We have developed a common data protocol (information being 
collected on all clients) at each site. The goal of the system is to enable us to describe the 
clients and the services they receive. Information in the system will include both client- 
specific clinical information (e.g., history, functional status, family information) and 
information about the provision of services. System components and their utilization status 
within the system will be derived from these data, along with financial data about the 
system. 

The data system will provide information to answer several important questions about 
the demonstration. Costs of services can be inferred by obtaining data from local 
providers on average costs for services. The system will also yield counts of "use" of 
each of the involved systems and will supplement more qualitative information collected by 
interviews. The system will enable verification of the involvement of multiple agencies in 
service provision and will identify the characteristics of the treatment team. Data from the 
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system will enable us to compile reports describing and comparing the sites. Examining 
client records for all of the children at a given site, will also enable us to determine 
characteristics of the children being served: ages, diagnoses, family constellation, history 
of involvement with different systems. 

Case conferences. One of the most difficult problems is how to evaluate clinical 
outcome in the context of a systems change effort. That scores on a measure of 
psychopathology change or do not change is not a very sensitive measure to assess the 
quality of care to chronically disturbed children. Seriously disturbed children may appear 
quite dysfunctional even when appropriate treatment has been provided; as well, while a 
child's symptoms may be greatly reduced, for example, in response to intensive 
intervention, the level of symptomatology at one point says little about the child's eventual 
adaptation to the community. Moreover, psychopathology measures fail to identify the link 
between provision of services and change. Although randomized clinical trial designs 
might allow inferences about these matters, they are particularly problematic when uiied in 
quasi -experimental research such as the present study. 

As an alternative, we are developing (in collaboration with the Yale University 
Child Study Center) a procedure to conduct and analyze case conferences on a sample of 
clients served by each project (see Table 5). Our goal is to develop a "picture" of how the 
systems implemented by each of the project sites work in practice. Our method involves 
gathering all of the members of the service team for a particular child and asking them to 
review systematically the diagnostic evaluation, the interventions and the outcomes of 
intervention. The procedure is designed to capture a rich portrait of the children treated 
by the project and understand the process by which providers decided upon specific 
interventions. 

These case conferences will include the presentation of a capsule version of a 
traditional case history as well as a consideration of the completeness of treatment 
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providers' evaluation of the child (e.g., "Was the treatment sensitive to issues of race, 
ethnicity and social class?", or "Does the diagnosis adequately describe the problem?"). 
Clinical team members will be asked to assess the treatment process as well, considering 
what treatments at which "doses" children received and what treatments may have been 
preferred but were not available. Patterns of service utilization and associated costs and 
their relationship to clients' needs will be examined longitudinally. The resulting data will 
also be used to understand the ways in which organizational and financial structures, 
policies and regulations enable or impede the provision of those services in individual 
cases. 

Conclusions 

Researchers are adept at identifying problems and describing theoretical solutions. 
We are, however, perhaps constitutionally incapable of helping practitioners solve specific 
problems. I will not disappoint those of you who share this view of researchers by 
offering much in the way of concrete advice. 1 would, nevertheless, like to make some 
specific observations about what we know and do not know about mental health services 
for children. It is too early to tell you much about what we have teamed directly from the 
RWJF demonstration, but my experience developing this research has helped me clarify 
the essential questions. 

• Perhaps, most importantly, we already know that treatment is better than 
no treatment. With confidence one can reject the null hypothesis and say 
that it is better for an emotionally disturbed child to receive professional 
mental health interventions than not Substantial evidence demonstrates that 
interventions do help children and that children, even with serious problems, 
can be helped. We do not, however, know for which children and under 
what conditions treatment is most beneficial. 

• We also know that inappropriate treatment (e.g., too restrictive, lacking 
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in intensity) is bad. Some children do not need to be in hospitals. Other 
children with serious disorders, left in dysfunctional environments without a 
supportive adult, need to be in some form of residential care. We do not, 
however, yet have a well-tested protocol for determining the appropriateness 
of an environment for a child. 

• We also know that fragmented treatment - where, for example, treatment 
provided in a residential setting is not followed by aftercare, is poor. But, 
we do not know how integrated treatment has to be. Should, in fact, there 
be a single treatment team and is it poor treatment, by definition, when the 
same person who provides therapeutic interventions is also responsible for 
decisions about the length and site of treatment. 

With each of these issues, how we choose to answer the question is crucial. No 
longer can we be concerned with purely clinical outcomes (i.e., impact on 
psychopathology), but we must be concerned with resources and resource use. Long-term 
psychotherapy may be effective, but may be far less cost-effective than other types of 
interventions. The problem becomes even more complex when erne knows that treatment 
is inappropriate, or is less effective than an alternative, but the treatment being offered is 
the best that one can offer. Better to have a child who is out of control in a locked 
residential setting than not getting any treatment. Better to see the child once a week, than 
not at all. 

To fix the current system we need to bridge the gap between theoretical and 
empirical knowle H «»e. We need, for example, to know how to operationalize coordination 
and be able to predict the effects of case management. We need to provide guidance on 
what kinds of systems of care will be effective and how care can be delivered efficiently. 
In sum, we need to use both our clinical wisdom and research skepticism if we are to play 
a role in restructuring mental health care for children and for making critical decisions 
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about allocating resources. 

I am convinced that reform of mental health services for children can proceed only 
if there is a strong alliance among advocates, practitioners, and researchers. Advocates 
need to goad those who control resources to develop new programs and end our society's 
shameful neglect of children. Practitioners need to consider new ways of providing 
services and work with advocates to insure that innovation is the hallmark of children's 
mental health care. Researchers need to collaborate with both of these groups to insure 
that we can learn from innovations in care. We can go from rhetoric about the need for 
better services to a more effective system, but it will require more than the resources 
generated by advocacy and the good will and efforts of mental health professionals. 

New knowledge is critical to this enterprise. There is no shame in our admitting 
that while we know the general direction that innovation must take, our specific ideas are 
tentative. As a researcher, I know all too well why the collaboration that I view as essen- 
tial has not worked and wliy conflict has replaced creative tension. Too often, researchers 
have attempted to impose methods that are not appropriate to the questions, while 
practitioners have been unwilling to view what they do as testable. At least in the case of 
children with serious emotional disorders, we need both innovation and research to help us 
develop a system that serves children, their families, and society. 

Each of us separately ~ advocates, professionals and researchers - has now gotten 
the attention of policy makers. For researchers, the adoption by NIMH of the National 
Plan for Research on Child and Adolescent Disorders is a palpable sign that our work is 
recognized as critical. I hope that the re-creation of NIMH as an NTH Institute, and the 
divorce of its service programs (to be adopted by a new "family" with the acronym 
ADAMHSA) does not symbolize the termination of government-sponsored collaboration 
between those who do and those who study what others do. There really is no alternative. 

A final comment about the role of psychology. As we develop a new system of 
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care, psychologists have a particularly important role to play. Multiple players are 
jockeying tor dominance in dealing with children mental health - psychiatrists, 
economists, policy experts, and even lawyers. As a profession with our "feet" (if not our 
heads) firmly planted in both science and practice, it is imperative that we play a central 
role. For the sake of our children, and the families who struggle witt the most difficult 
problems of life, I hope none of you is dissuaded by the complexity of the problem. 
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FIGURE 1 . MHSPY DEMONSTRATION SITES 
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TABLE 1 . RANGE OF SELECTED SERVICES AT REPRESENTATIVE MHSPY DEMONSTRATION SITES 



SERVICE 


Ohio 


Pennsylvania 


Vermont 


Wisconsin | 


Outpatient therapy 


0 


O 


0 


0 


Emergency stabilization 


0 


0 


0 


0 


1 Crisis outreach 


0 


X 


X 


X 


I Interagency case management ( > 1:15 ) 


0 








I Intensive case management ( 1:15 ) 




0 


X 


0 


| Respite care 




0 


0 




8 Intensive home-based services 


0 


0 


0 


0 


| Therapeutic foster care 


0 


X 


0 


° I 


I Transitional living programs 


X 


0 


0 


0 | 


Group residential care 


0 


0 


0 


0 | 


Intensive residential treatment 


0 


0 


0 


o I 


Hospitalization 


0 


0 


0 


0 


Day treatment 


0 


0 


0 


0 


Wrap-around services 


X 


0 


0 




Parent support network 




0 


X 


x I 



O - In place at beginning of MHSPY project. 
X - Being put in place as part of MHSPY project. 
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TABLE 2. MHSPY EVALUATION RESEARCH QUESTIONS 





ORGANIZATIONS 


FINANCES 


CLIENTS 1 


J CONTEXT 


Socio-demographic features 
Prior historv of chanoe 
Components of system 
Interagency structures 
Services available 


Public and private resources 

Lcyidiauuii fli ivi 1 tTyillallUi !o 

Other factors which restrict: 

- service provision 

- financing of services 


Socio-demographic profile 1 
runcuonai cnaraciensucs | 
Profile of treatment history I 
Profile of service system use 1 


1 PROCESS 


Interagency relationships 
Coordination of services 
Role of case managers 


Coordination of resources 
Blending of resources 
Resource allocation 


»*— « 

Services not available 
Service/heed match 


I OUTCOME 


Changes in system 
Changes in coordination 
Impact of case management 


Changes in financing 
Cost of services 
High-/low-cost case profiles 


Changes in adequacy of care 
Changes in functional abilities 
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TABLE 3. ORGANIZATIONAL ASSESSMENT 



From: Interviews, Surveys, 
Document Review 



Sample: Direct Service Providers, 
Administrators, Parents 



Change Plans 

History and context of change effort 

Strategies for managing and implementing change 

Organizations 

Components of the service system 
Interagency structures, mechanisms and relationships 
Range and capacity of available and oroposed services 
Coordination of services 

Case management and it's relationship to other service provision 
Finances 

Funding stream shifts and reallocations 
New resources to be added to system of care 

Clients 

Socio-demographic characteristics of general population 
Definition of the target population 
Points of entry to the service system 
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TABLE 4. MANAGEMENT INFORMATION SYSTEM 
From: SEDTRACK/QUICKTRACK Sample: All MHSPY Clients 



Referral 

Socio-demographic information 
Clinical status and history 

Relevant medical, educational, social service, juvenile justice 
status and history 

Relevant status and history of family members 

Ran of Care 

Issues and goals 
Services needed 

Assignment of financial and service provision responsibility 

Treatment 

Significant events 
Changes in Plan of Care 
Services received 

Organizational components providing services 

Units of service received 

Cost per unit of service received 



TABLE 5. CASE CONFERENCE 
From: Clinical Case Conferences Sample: Sample of MHSPY Clients 

Availability and organization of resources at conference 

Difficulties/needs of child 

Strengths and weaknesses of adult care givers and organizations 

Optimization of clinical care 
Treatment alliances 
Coordination of services 
Flexibility of services/treatment planning 
Availability of resources 
Adequacy of monitoring 
Adequacy of care 
Adequacy of planning 
Process of consensus within conference 

Adequacy of treatment team's work 



